Generalization in Portfolio-Based Algorithm Selection

Portfolio-based algorithm selection has seen tremendous practical success over the past two decades. This configuration procedure works by first selecting a portfolio of diverse parameter settings, and then, on given problem instance, using an selector to choose setting from with strong predicted performance. Oftentimes, both are chosen training set typical instances application domain at hand. In this paper, we provide provable guarantees for portfolio-based selection. We analyze how large should be ensure that resulting selector's average performance is close its future (expected) involves analyzing three key reasons why these quantities may diverge: 1) learning-theoretic complexity selector, 2) size portfolio, 3) algorithm's as function parameters. introduce end-to-end analysis construction together. prove if large, overfitting inevitable, even extremely simple selector. With experiments, illustrate tradeoff exposed our theoretical analysis: increase size, can hope include well-suited every possible but it becomes impossible avoid overfitting.